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CFO 17857 US 



INFORMATION PROCESSING METHOD 
AND IMAGE REPRODUCTION APPARATUS 

BACKGROUND OF THE INVENTION 
5 Field of the Invention 

The present invention relates to a method and . 
an apparatus which set, in a virtual space which is 
constructed based on an actually taken image, a 
position of an annotation to be displayed on the 

10 actually taken image. 
Related Background Art 

An attempt at taking (or shooting) a real space 
by a camera mounted on a vehicle (or a movable body), 
and representing the taken real space as a virtual 

15 space based on taken real image data by using a 
computer has been proposed. For example, Endo, 
Katayama, Tamura, Hirose, Watanabe, Tanikawa: 
"Computer Visualization Of Cybercities Using Vehicle- 
Mounted Cameras", Society Conference of IEICE 

20 (Institute of Electronics, Information and 

Communication Engineers), PA-3-4, pages 276-277, 1997, 
or Endo, Katayama, Tamura, Hirose, Watanabe, 
Tanikawa: Building Image-Based Cybercities By Using 
Vehicle-Mounted Cameras (2) -Generation Of Wide-Range 

25 Virtual Environment . By Using Photo-Realistic Images-, 
Proc. of the Virtual Reality Society of Japan, Volume 
2, pages 67-70 (1997.9) should be referred. 
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Incidentally, as a method of representing a 
taken real space as a virtual space baised on data 
representing an actually taken image (hereinafter 
called actually taken image data) , there is a method 
5 of reproducing a geometrical model of the real space 
from the actually taken image data and representing 
the reproduced model in conventional CG (Computer 
Graphics) technique. However, in this case, there 
are limits in accuracy of the model and truth to 

10 nature of the model. On one hand, IBR (Image-Based 
Rendering) technique of representing a virtual space 
by using the actually taken image without reproducing 
any geometrical model attracts attention in recent 
years. Here, because the IBR technique is based on 

15 the actually taken image, a realistic virtual space 

can be represented. Besides, although vast times and 
efforts are necessary to form the geometrical model 
which covers a vast space such as city and town, such 
time and effect are unnecessary in the IBR technique 

2 0 because any geometrical model is not reproduced. 

To structure a virtual space which enables 
walk-through by using the IBR technique, it is 
necessary to generate and present an image according 
to the position of an experiencing person (also cited 

25 as an observer hereinafter) in the virtual space. 
For that purpose, in a system of this kind, each 
image frame of the actually taken image data is 
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correlated with the position within the virtual space 
and stored in advance, the corresponding image frame 
is obtained based on the position and sight line 
direction of the experiencing person in the virtual 
5 space, and the obtained image frame is reproduced. 
Incidentally, in order to enable the 
experiencing person to see a desired direction at 
each viewpoint position during the walk-through 
operation within the virtual space, the image frame 

10 corresponding to each viewpoint position is stored in 
advance as a panoramic image which covers the range 
wider than an angle of view at a time when the image 
at the viewpoint position in question is reproduced. 
That is, when the image in question is reproduced, 

15 the stored panoramic image is read based on the 

viewpoint position of the experiencing person within 
the virtual space, a partial image is cut out from 
the read panoramic image on the basis of the sight 
line direction of the observer, and the cut-out image 

20 is then displayed. When the trail of the viewpoint 
position within the virtual space is the same as the 
trail of the vehicle on which the camera is mounted, 
the observer feels as if the observer oneself takes 
the vehicle and runs. 

25 Moreover, by synthesizing an annotation such as 

a name or the like of, e.g., a building to the 
building in question included in the image and 
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displaying the synthesized annotation together with 
the image of the building in question, it is possible 
to provide more expressive information to the 
observer. Furthermore, by displaying such an 
5 annotation, a marker or a sign which is obscure 
because the actually taken image is dark can be 
clearly known and grasped by the observer. 

When the virtual space is described and 
represented by using the geometrical model, the 

10 annotation can be synthesized and displayed at a 

desired position on the image. On one hand, when the 
virtual space is constructed in the IBR technique in 
which any. geometrical model is not used, it is 
necessary to determine the display position of the 

15 annotation in regard to each image. 

However, conventionally, when the annotation is 
synthesized and displayed in the above virtual space 
which has been constructed in the IBR technique, it 
is necessary for the user to. manually determine the 

20 annotation display position in regard to each image, 
whereby it takes a lot of trouble with working in 
determining the annotation display position when 
there are a large number of images. 

25 SUMMARY OF THE INVENTION 

The present invention has been made in 
consideration of such a conventional problem, and an 
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object thereof is to simplify an operation for 
determining an annotation display position. 

In order to achieve the above object, the 
present invention is characterized by an information 
5 processing method comprising: a viewpoint 

position/sight line direction determination step of 
determining a viewpoint position and a sight line 
direction on a map; an annotation display position 
determination step of determining an annotation 

10 display position of an object, from the position of 

the object in question on the map determined based on 
observation directions of the object in question in 
plural panoramic images, the viewpoint position, and 
the sight line direction; and a synthesis step of 

15 synthesizing an annotation image to the annotation 
display position on an actually taken image 
corresponding to the viewpoint position. 

Moreover, the present invention is 
characterized by an information processing method, 

20 used in an image reproduction apparatus for achieving 
walk-through in a virtual space represented by using 
an actually taken image, of synthesizing an 
annotation image to the^ actually taken image, the 
method comprising the steps of: setting an annotation 

25 display position in each of the plural actually taken 
images; calculating an annotation display position to 
another actually taken image located between the 
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plural actually taken images, by using the annotation 
display positions respectively set in the plural 
actually taken images; and synthesizing the 
annotation image to the actually taken image on the 
5 basis of the calculated annotation display position. 

Other features and advantages of the present 
invention will be apparent from the following 
description taken in conjunction with the . 
accompanying drawings, in which like reference 
10 characters designate the same or similar parts 
throughout the figures thereof. 

BRIEF DESCRIPTION OF THE DRAWINGS 

Fig. 1 is a block diagram showing the 
15 functional structure of a walk-through system 

according to the embodiment of the present invention; 
Fig. 2 is a block diagram showing the hardware 

structure of an image reproduction apparatus .1 

according to the embodiment of the present invention; 
20 Fig. 3 is a diagram for explaining a 

representation method of a virtual space according to 

the embodiment of the present invention; 

Fig. 4 is a diagram showing an example of 

attributes of section points and routes; 
25 Fig. 5 is a diagram for explaining 

correspondence between a panoramic image and a 

direction of the route; 
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Fig. 6 is a diagram for explaining an 
annotation display position determination method in 
an annotation display position determination unit 50; 

Fig. 7 is a flow chart for explaining an 
5 operation of the annotation display position 
determination unit 50; 

Fig. 8 is a diagram for explaining an 
annotation synthesis process by an image reproduction 
control unit 40; 
10 Fig. 9 is a diagram for explaining a method of 

determining an object position based on two panoramic 
images; 

Fig. 10 is a flow chart for explaining a 
procedure to determine the position of each object on 
15 a map based on the two panoramic images; 

Fig. 11 is a diagram showing a GUI (graphical 
user interface) 1000 for determining the object 
position; 

Fig. 12 is a diagram for explaining a method of 
20 determining the object position on the GUI 1000; 

Fig.- 13 is a diagram showing a map on which an 
object to which an annotation is intended to be 
displayed, section points, and routes are disposed; 
Fig. 14 is a diagram showing an example of 
25 attributes of the object to which the annotation is 
displayed; 

Fig. 15 is a diagram showing a GUI 2000 for 
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setting an annotation display position in units of 
panoramic image; 

Fig, 16 is a diagram for explaining a method of 
determining the annotation display position in units 
5 of panoramic image on the GUI 2000; 

Fig. 17 is a diagram showing attributes of an 
object to which an annotation is intended to be 
displayed, according to the third embodiment; 

Fig. 18 is a flow chart for explaining a 
10 procedure to determine an annotation display position 
in units of panoramic image, according to the third 
embodiment;, 

Fig. 19 is a flow chart for explaining the 
procedure to determine the annotation display 
15 position in units of panoramic image, according to 
the third embodiment; 

Fig. 20 is a flow chart for explaining a 
procedure to determine an annotation display position 
for a certain object, in the annotation display 
20 position determination unit 50; 

Fig. 21 is a diagram for explaining a method of 
determining an annotation display position according 
to the fourth embodiment; 

Figs. 22A, 22B and 22C are diagrams for 
25 explaining relations of object observation directions 
01 and 92 from respective panoramic images at two 
points, and an object observation direction 0i from 
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the panoramic image on a route located between the 
two points; and 

Figs. 23A, 23B, 23C, 23D, 23E, 23F and 23G are 
diagrams for explaining respective relations of frame 
5 numbers and annotation display positions, based on 
the object observation directions 01 and 02. 

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS 

Hereinafter, the preferred embodiments of the 

10 present invention will be explained with reference to 
the attached drawings . 
(First Embodiment) 

Initially, a walk-through system in a virtual 
space according to the first embodiment of the 

15 present invention will be explained. In the present 
embodiment, panoramic image data is generated from 
actually taken image data obtained by plural cameras 
(or shooting devices) mounted on a vehicle such as an 
automobile or the like, the generated panoramic image 

20 data is correlated with positions on a map 

corresponding "to respective positions in a real space, 
and the correlated data are together stored. Then, a 
display image is generated based on the stored 
. panoramic image data in accordance with a viewpoint 

25 position (i.e., the position on the map) and a sight 
line direction of an experiencing person (or an 
observer) , thereby achieving walk-through in the 
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virtual space. 

Fig. 1 is a block diagram showing the 
functional structure of the walk-through system 
according to the present embodiment. An image 
5 reproduction apparatus 1 which constitutes the walk- 
through system is equipped with an operation unit 10, 
a viewpoint position/sight line direction 
determination unit 20 , a map data storage unit 30, an 
image reproduction control unit 40, an annotation 

10 display position determination unit 50, an annotation 
data storage unit 60, an image data storage unit 70, 
and a display unit 80. 

Fig. 2 is a block diagram showing the hardware 
structure of the image reproduction apparatus 1 

15 according to the present embodiment. Here, it should 
be noted that the hardware structure shown in Fig. 2 
is equivalent to that of an ordinary personal 
computer. In Fig. 2, a disk 105 acts as the image 
data storage unit 70, and also acts as the map data 

20 storage unit 30 and the annotation data storage unit 
60. 

A CPU 101 functions as the viewpoint 
position/sight line direction determination unit 20, 
the image reproduction control unit 4 0 and the 
25 annotation display position determination unit 50 by 
executing programs stored in the disk 105, a ROM 106 
and/or an external memory (not shown) . 
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Moreover, the CPU 101 issues various display 
instructions to a CRTC (cathode ray tube controller) 
102, whereby desired display is achieved on a CRT 104 
by the CRTC 102 and a frame buffer 103. Here, 
5 although the CRTC 102 and the CRT 104 are shown 

respectively as a display controller and a display in 
Fig. 2, the present invention is not limited to this. 
That is, instead of the CRT, an LCD (liquid crystal 
display) or the like can be of course used as the 

10 display. Incidentally, the CRTC 102, the frame 
buffer 103 and the CRT 104 together act as the 
display unit 80. Besides, a RAM 107 is provided as a 
working memory for the CPU 101 and the like. 

A mouse 108, a keyboard 109 and a joystick 110 

15 which are used by a user to input various data and 
information to the image reproduction apparatus 1 
together act as the operation unit 10. 

Next, a schematic operation of the image 
reproduction apparatus 1 in the walk-through system 

20 according to the present embodiment will be explained 
The operation unit 10 which is equipped with 
the mouse, the keyboard, the joystick and the like is 
used to generate a movement parameter of the 
viewpoint position and a rotation parameter of the 

25 sight light direction. In the present embodiment, 
although the joystick 110 is used to control the 
viewpoint position and the sight line direction, 
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another input device such as a game controller or the 
like may be used- Incidentally , the inclination 
angle and the rotation angle of the joystick 110 can 
be controlled independently- In the present 
5 embodiment, the operation to incline the joystick 110 
corresponds to the movement of the viewpoint position 
in the virtual space, and the operation to rotate the 
joystick 110 rightward and leftward corresponds to 
the rotation of the sight line direction. 

10 Incidentally, the map data storage unit 30 

stores therein two-dimensional map image data. 

Moreover, the viewpoint position/sight line 
direction determination unit 20.. determines the 
viewpoint position and the sight line direction of 

15 the observer on the map image represented by the two- 
dimensional map image data stored in the map data 
storage unit 30, on the basis of the movement 
parameter and the rotation parameter input through 
the operation unit 10. 

2 0 Furthermore, the image data storage unit 7 0 

stores therein the panoramic image data corresponding 
to each position on the map. Here, it should be 
noted that the image data storage unit 70 need not 
exist as a local device, of the image reproduction 

25 apparatus 1. That is, it is possible to provide the 
image data storage unit 70 on a network, and thus 
read the image data from the image data storage unit 
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70 through the network. 

The image reproduction control unit 40 receives 
the data concerning the viewpoint position and the 
sight line direction of the observer on the map from 
5 the viewpoint position/sight line direction 

determination unit 20, and . then reads the image data 
corresponding to the view point position from the 
image data storage unit 7 0 based on the received data. 
Incidentally, in the present embodiment, to correlate 
10 the viewpoint position on the map with the image data, 
the necessary data have the following data storage 
formats . 

That is, it is assumed that the movement of the 
observer is limited only on a taking (shooting) route, 

15 the route is partitioned by section points such as an 
intersecting point (diverging point) , a corner and 
the like, and the route is represented as the section 
points and the route located between the two section 
points. The section points are set on the two- 

20 dimensional image, and the route is the line segment 
located between the section points. Then, an ID 
(identification) is added to each of the section 
points and the routes, the panoramic image taken at 
the position in the real space is assigned to the 

25 corresponding section point, and the panoramic image 
group between the panoramic images respectively 
- assigned to the section points of both the ends of 
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the route is assigned to the route in question. Fig. 
3 shows such an aspect. That is, in Fig. 3, the ID 
of Rl is given to the line segment (route) located 
between the section point of which the ID is CI. and 
5 the section point of which the ID is C2 . Then, in a 
case where the panoramic images respectively 
corresponding to the section points CI and C2 are 
specified based on GPS (Global Positioning System) 
data or the like, the panoramic image of which the 

10 frame number is n is assigned to the section point CI, 
and the panoramic image of which the frame number is 
(n + m) is assigned to the section point C2 . After 
the panoramic images were assigned to the respective 
section points, the panoramic image group including 

15 the panoramic images of which the frame numbers are 
(n + 1) to (n + m - 1) is automatically assigned to 
the route Rl . Similarly, the respective panoramic 
image groups are assigned to the routes R2 to R5 
respectively. 

20 Incidentally, as shown in Fig. 4, each of the 

section points and the panoramic images has the two- 
dimensional coordinates on the map as its attribute. 
Here, although the two-dimensional coordinates on the 
map are generally calculated from the latitude and 

25 longitude data obtained based on the GPS data, the 
two-dimensional coordinates may be obtained from 
image information through a computer vision. 
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Moreover, it is possible to obtain the two- 
dimensional coordinates of only the section points at 
both the ends of the route based on the latitude and 
longitude data, and it is further obtain the two- 
5 dimensional coordinates of the panoramic images on 
the route between these section points through an 
interpolation operation. 

The image reproduction control unit 4 0 gives 
the viewpoint position on the map to the annotation 

10 display position determination unit 50. Then, the 
annotation display position. determination unit 50 
determines the display position of the annotation 
based on the given viewpoint position information, 
and gives the determined annotation display position 

15 to the image reproduction control unit 40. How to 
determine the annotation display position will be 
described later. After then, the image reproduction 
control unit 40 cuts out the panoramic image 
according to the angle of view displayed on the 

20 display unit 80, performs projection conversion to 
the cut-out panoramic image, synthesizes the 
annotation image to the converted panoramic image in 
accordance with the annotation display position, and 
then generates the image to be displayed on the 

25 display unit 80. 

Subsequently, the display unit 80 displays the 
image generated by the image reproduction control 
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unit 40. 

Next, the operation of the annotation display 
position determination unit 50 will be explained in 
detail. In the present embodiment, on the route 
5 located between the section points as shown in Fig. 5, 
it is assumed that the front direction of the 
panoramic image is in parallel with the direction of 
the route in question, i.e., the camera forwarding 
direction in the image taking or shooting. 

10 Fig. 6 is a diagram for explaining an 

annotation display position determination method to 
be performed by the annotation display position 
determination unit 50. For simplicity, it is assumed 
that the section point CI is the origin of an xy 

15 plane, and the section point C2 is set on the x axis 
of this plane (that is, the route Rl constitutes a 
part of the x axis) . 

In Fig. 6, the coordinates of the section point 
CI, the section point C2 and a building (object) A to 

20 which the respective annotations are intended to be 

displayed on the map are respectively (0, 0), (x2, 0) 
and (xo, yo) . Moreover, in the panoramic image 
corresponding to the section point CI, the horizontal 
position (i.e., position' in horizontal direction) at 

25 which the annotation of the building A is displayed 
is represented in a relative angle 91 (radian) from 
the front direction of the panoramic image, as 
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15 



20 



25 



follows . 

r arctan (yo/xo) (xo ^ 0) 
01 = j n/2 (xo = 0, yo > 0) 
^-n/2 (xo =0, yo < 0) 
Furthermore, in the panoramic image 
corresponding to the section point C2, the horizontal 
position at which the annotation of the building A is 
displayed is represented in a relative angle 92 
(radian) from the front direction of the panoramic 
image, as follows. 

/arctan{yo/ (xo-x2) } (xo x2 ) 
92 = ■ n/2 (xo = x2, yo > 0) 
(xo = x2, yo < 0) 
Similarly, in the panoramic image corresponding 
to the point (x, 0) on the route Rl, the horizontal 
position at which the annotation of the building A is 
displayed is represented in a relative angle 9 
(radian) from the front direction of the panoramic 
image, as follows. 

/ arctan{yo/ (xo-x) } (xo + x) 



9 = \ n/2 (xo = x, yo > 0) 
-n/2 (xo = x, yo < 0) 
The annotation display position determination 
unit 50 determines the horizontal positions at which 
the annotations are displayed, in accordance with the 
above formulae. Fig. 7 is a flow chart for 
explaining the operation of the annotation display 
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position determination unit 50. In Fig. 7, in a step 
S101, new viewpoint information (i.e., the viewpoint 
position and the sight line direction) is first 
obtained. Then, it is judged in a step.S102 whether 
5 or not the route determined based on the new 

viewpoint information obtained in the step S101 is 
the same as the route in the previous frame . When it 
is judged that the route determined based on the new 
viewpoint information is the same -as the route in the 

10 previous frame, the flow advances to a step S105. On 
the contrary, when it is judged in the step S102 that 
the route determined based on the new viewpoint 
information is a new route different from the route 
in the previous frame, the flow advances to a step 

15 S103. In the step S103, the object to which the 

annotation is displayed is determined on the route in 
question. In the present embodiment, it should be 
noted that the annotations can be respectively 
displayed to the plural objects. After the object to 

20 which the annotation is displayed was determined in 
the step S103, the flow advances to a step S104. In 
the step S104, one of the section points at both the 
ends of the route in question is set as the origin of 
the xy plane, the coordinate axis is rotated so that 

25 the route in question coincides with the x axis, and 
the relative positions of all the objects to which 
the annotations are respectively displayed are 
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calculated. Next, in the stepS105, an annotation 
display position 9 (i.e., a relative angle from the 
front direction of the panoramic image) in the 
panoramic image corresponding to the viewpoint 
5 position in question is obtained by the above formula,, 
in regard to each of all the objects to which the 
annotations are respectively displayed. After then, 
it is judged in a step S106 whether or not to end the 
operation. When the operation should be continued, 

10 the flow returns to the step S101 to again obtain new 
viewpoint information . 

Fig. 8 is a diagram for explaining an 
annotation synthesis process by the image 
reproduction control unit 40. When the annotation 

15 display position 9 is determined by the annotation 
display position determination unit 50, the image 
reproduction control unit 40 cuts out the panoramic 
image according to the sight line direction and an 
angle of view a. Then, the annotation image read 

20 from the annotation data storage unit 60 is 

synthesized on the cut-out panoramic image, whereby 
the display image is finally generated. Here, it 
should be noted that clinographic conversion for 
converting the panoramic image into a perspective 

25 projection image is performed only to the panoramic 
image . 

As described above, according to the first 
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embodiment, the annotation display .position is 
determined based on the coordinates of the object 
position to which the annotation is intended to be 
displayed and the viewpoint position of the observer 
5 on the two-dimensional map, whereby it is possible to 
. achieve saving of work and time when the annotation 
display positions are determined to a large number of 
images . 

(Second Embodiment) 

10 In the above first embodiment, the annotation 

display position is determined based on the 
coordinates, on the map, of the object position to 
which the' annotation is intended to be displayed and 
the viewpoint position of the observer. On one hand, 

15 in the second embodiment, the position of an object 
on the map is determined based on the observation 
directions of that object in two panoramic images, 
whereby an annotation can be displayed at an 
appropriate position even if accuracy of the 

20 coordinates of the object position and the sight line 
direction on the map is low. 

Fig. 9 is a diagram for explaining a method of 
determining the object position based on the two 
panoramic images ^ In the present embodiment, the 

25 position of the object on the map is determined based 
on the coordinates of the viewpoint position on the 
map, and moreover the position of the object on the 
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map is determined in regard to each route. 
Incidentally, when the annotation display position is 
calculated, the position of the object on the map 
determined on the route where the. viewpoint position 
5 exists is used. In Fig. 9, for simplicity, it is 

assumed that a section point CI is the origin of the 
xy plane, and a section point C2 is set on the x axis 
of this plane (that is, a route Rl constitutes a part 
of the x axis) . Moreover, the coordinates of the 
10 section points CI and C2 are (0, 0) and (x2, 0) 
respectively, and the front direction of the 
panoramic image on the route Rl always corresponds to 
the positive direction of the x axis. 

When the observation direction (i.e., the 
15 relative direction from the front direction) from the 
section point CI of an object (a building) A to which 
the annotation is intended to be displayed is 91 and 
. the observation direction from the section point C2 
is 92, the coordinates (xo, yo) of the object on the 
20 map can be obtained from following formulae. 

/ 0 (91 = n/2, -n/2) 

xO = j x2 (92 = n/2, -n/2) 

Ix2tan92/ (tan92 - tanGl) (other). 
f -x2tan92 (91 = n/2, -n/2) 

x2tan91 (92 = n/2, -n/2) 

( x2tan91tan92/ (tan92 - tan91) (other) 
Fig. 10 is a flow chart for explaining a 



25 yO = 
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procedure to determine the position of the object on 
the map based on the two panoramic images . Initially, 
in a step S201, the object to which the annotation is 
intended to be displayed is determined. Next, it is 
5 judged in a step S202 whether or not the object 
position determination ends on all the routes to 
which the annotation of the object in question is 
displayed. When it is judged that the object 
position determination does not end on all the routes, 

10 the flow advances to a step S203 to determine the 
route to which the object position determination 
should be performed. Then, in a step S204, the 
object position on that route is calculated by the 
above formulae. On the contrary, when it is judged 

15 in the step S202 that the object position 

determination ends on all the routes, the flow 
advances to a step S205. Then, it is judged in the 
step S205 whether or not the position determination 
ends to all the objects to which the annotations are 

20 intended to be displayed. When it is judged that the 
position determination ends to all the objects, the 
position determination ends. On the contrary, when 
it is judged that the position determination does not 
end to all the objects, the flow returns to the step 

25 S201 to determine the next object. 

Fig. 11 is a diagram showing a GUI 1000 for 
determining the object position in the present 
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embodiment. The GUI 1000 includes a map display 
window 1010 for displaying a two-dimensional map 
image, panoramic image display windows 1020 and 1021 
for displaying panoramic images corresponding to the 
5 section points at both the ends of the route selected 
on the map display window 1010, an object addition 
button 1030, an existing object button 1040, and an 
update button 1050. 

When the position determination is performed to 

10 a new object to which any position determination is 

not yet performed, the object addition button 1030 is 
clicked by a mouse. On one hand, when the position 
of the object to which the position determination has 
been performed is corrected, the existing object 

15 button 1040 is clicked by the mouse to select the 
desired object from the list of the objects to be 
displayed . 

The map display window 1010 displays the two- 
dimensional map image on which the section points and 

20 the routes are displayed. When the user clicks by 
the mouse the route to which the object position is 
intended to be determined, the panoramic images 
respectively corresponding to the section points at 
both the ends of the selected route are displayed* 

25 respectively on the panoramic image display windows • 
1020 and 1021, - . 

Incidentally, it should be noted that the two 
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typical panoramic images may not be the panoramic 
images respectively corresponding to the section 
points at both the ends of the route. That is, the 
two typical panoramic images may be panoramic images 
5 at independent positions respectively designated on 
the map by the user. For example, panoramic images 
at the positions of the section points on the 
different routes may be used. 

Fig. 12 is a diagram for explaining a method of 

10 determining the object position on the GUI 1000. 
Here, a case where the position of the object 
(building) A on the route Rl located between the 
section points CI and C2 is determined will be 
explained. When the route Rl is clicked by the mouse, 

i5 the panoramic images corresponding to the respective 
section points CI and C2 are displayed on the 
panoramic image display windows 1020 and 1021 
respectively. First, on the panoramic image display 
window 1020, when the direction in which the object A 

20 is observed is clicked by the mouse, the straight 
line parallel with the vertical direction of the 
panoramic image passing the clicked point on the 
panoramic image display window 1020 is drawn, and the 
straight line indicting the clicked direction is 

25 drawn on the map display window 1010. Then, the 
similar operations are performed on the panoramic 
image display window 1021 and the map display window 
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1010. As a result, on the map display window 1010, 
the point at which the two straight lines intersect 
is calculated and obtained as the position of the 
object A on the route Rl . 
5 When the object position determination is 

performed on all the routes to which the annotations 
are intended to be displayed, the update button 1050 
is depressed to store the obtained position data. 

Fig. 13 is a diagram showing a map on which an 

10 object to which an annotation is intended to be 

displayed, section points, and routes are disposed. 
In Fig. 13, the object (building) A can be observed 
from routes Rl, R2, R3 and R4 . Thus, when the 
annotation of the object A is displayed on the routes 

15 Rl and R2, the position of the object A on the route 
Rl is calculated from the panoramic images 
corresponding to the section points CI and C2, and 
the position of the object A on the route R2 is 
calculated from the panoramic images corresponding to 

20 the section points C2 and C3 . Fig. 14 is a diagram 
showing an example of attributes of the object to 
which the annotation is displayed. That is, the 
position coordinates (xol, yol) of the object on the 
map are used when the annotation display position on 

25 the route Rl is determined, and the position 

coordinates (xo2, yo2) of the object on the map are 
used when the . annotation display position on the 
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route R2 is determined. Here, it should be noted 
that the annotation image can be made different in 
regard to each route. 

Incidentally, the annotation image is given as 
5 an image according to a JPEG (Joint Photographic 

Experts Group) format in Fig. 14. However, another 
image format may be of course used, and besides, a 
moving image may be used as the annotation image. 
Then, the annotation display position 

10 determination unit 50 determines the position at 
which the annotation is displayed, by using the 
position coordinates of the object determined as 
above on the map. 

As described above, according to the second 

15 embodiment, because the position of the object on the 
map is determined based on the observation directions 
of the object in question in the two panoramic images, 
the annotation can be displayed at the appropriate 
position even if accuracy of the coordinates of the 

20 object position and the sight line direction on the 
map is low. 

Moreover, because the GUI is used, the position 
of the object on the map can be easily determined. 
(Third Embodiment) 
25 In the above second embodiment, the position. of 

the object to which the annotation is intended to be 
displayed on the map is determined based on the 
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observation directions of the object in question in 
the two panoramic images. On one hand, in the third 
embodiment, it enables to set an annotation display 
position in units of panoramic image and 
5 preferentially use the set annotation display 

position, thereby performing annotation display at a 
more appropriate position. 

Fig. 15 is a diagram showing a GUI 2000 for 
setting the annotation display position in units of 

10 panoramic image. In Fig. 15, because a map display 

window 1010, panoramic image display windows 1020 and 
1021, an object addition button 1030, an existing 
object button 1040 and an update button 1050 are 
respectively the same as those shown in Fig. 11, the 

15 explanations thereof will be omitted. Besides, a 
panoramic image display window 1022 is used to 
display the panoramic image corresponding to an 
arbitrary point on the selected route. 

Fig. 16 is a diagram for explaining a method of 

20 determining the annotation display position in units 
of panoramic image on the GUI 2000. Here, after the 
position of the object (building) A was determined 
from the observation directions of the panoramic 
images at the two section points, when a point on the 

25 route is clicked by the mouse, the panoramic image 
corresponding to the clicked point is displayed on 
the panoramic image display window 1022. At the same 
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time, the annotation display position determined from 
the position of the object A is represented as the 
straight line parallel with the vertical direction of 
the panoramic image. When accuracy of the position 
5 coordinates of the panoramic image on the map is low, 
there is a fear that the represented annotation 
display position is deviated or shifted from the 
position at which the annotation is intended to be 
actually displayed. Therefore, the appropriate 

10 annotation display position is clicked on the 

panoramic image display window 1022 to prevent this. 
As described above, the annotation display position 
determined in units of panoramic image is used in 
preference to the annotation display position 

15 determined from the position of the object on the map 
and the viewpoint position in the annotation display 
position determination unit 50. 

Fig. 17 is a diagram showing attributes of the 
object to which the; annotation is intended to be 

20 displayed, according to the third embodiment. That 
is, in regard to the object (the building A), the 
annotation display positions are set independently 
for the two panoramic images (frame numbers n and m) 
on the route- Rl . The independently set annotation 

25 display positions are described as relative angles 9n 
and 9m from the front direction of the panoramic 
image . 
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Fig. 18 is a flow chart for explaining a 
procedure to determine the annotation display 
position in units of panoramic image, according to 
the third embodiment. In Fig. 18, because the 
5 processes in steps S201 to S205 are respectively the 
same as those shown in Fig. 10, the explanations 
thereof will be omitted. Then, in Fig. 19, when it 
is judged in a step S301 not to determine the 
annotation display position in units of panoramic 

10 image, the flow returns to the step S202 (Fig. 18). 
On the contrary, when it is judged to determine the 
annotation display position in units of panoramic 
image, the flow advances to a step S302 to select and 
determine the panoramic image to which the annotation 

15 display position is determined. Next, in a step S303, 
the annotation display position is set in the 
panoramic image in question. After then, it is 
judged in a step S304 whether or not to end the 
operation. When the annotation display position is 

20 determined to another panoramic image, the flow 
returns to the step S302. 

Fig. 20 is a flow chart for explaining a 
procedure to determine the annotation display 
position for a certain object, in the annotation 

25 display position determination unit 50. First, when 
it is judged in a step S311 that the annotation 
display position for the certain object has been set 



in the panoramic image corresponding to the viewpoint 
position, the set annotation display position is used 
as it is. On the contrary, when it is judged that 
the annotation display position. for the certain 
5 object is not set, the annotation display position is 
determined from the object position and the viewpoint 
position in a step S312, and then the determined 
annotation display position is used. After then, the 
annotation display position is determined in a step 

10 S313, and the operation ends. 

As described above, according to the third 
embodiment, the annotation display position can be 
set in units of panoramic image, and the set 
annotation display position can be preferably used, 

15 whereby the annotation display can be performed at 
the more appropriate position. 

Moreover, the GUI is used, whereby the 
annotation display position can be easily set in 
units of panoramic image. 

2 0 (Fourth Embodiment) 

In the above first to third embodiments, the 
annotation display position is determined based on 
the observer's viewpoint position on the map and the 
object position on the map. On one hand, in the 

25 fourth embodiment, the annotation display position is 

easily determined without using a position on the map. 
Fig. 21 is a diagram for explaining a method of 
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determining the annotation display position according 
to the fourth embodiment. In Fig. 21, it is assumed 
that a route Rl beginning from a section point CI and 
ending to a second point C2 is represented by the 
5 straight line. Moreover, it is assumed that the 

front direction of a panoramic image corresponding to 
the section point CI, the front direction of a 
panoramic image corresponding to the section point C2, 
and the front directions of panoramic images * included 

10 in a group corresponding to the route Rl are all the 
same (i.e., the direction extending from the section 
point CI to the section point C2 ) . Furthermore, it 
is assumed that the panoramic image of which the 
frame number is n is related to the section point CI, 

15 the panoramic image of which the frame number is (n + 
m) (m > 0) is related to the section point C2, and 
the panoramic images of which the frame numbers are 
(n +1) to (n + m - 1) are related to the route Rl . 
When the annotation of a building (object) A 

20 shown in Fig. 21 is displayed to the group of the 

panoramic images related to the route Rl, observation 
angles 91 and 02 of the building A at the section 
points CI and C2 of both the ends of the route Rl are 
first obtained. Here, the observation angles 01 and 

25 02 are obtained beforehand in a preprocess in regard 
to each route . 

In the annotation display position 
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determination unit 50, when the observation 
directions of the building A from the section points 
CI and C2 are respectively given by the observation 
angles 91 and 92, an annotation display position 
5 (angle) 9i of the building A in the panoramic image 
of a frame number (n + i) (i > 0) on the route Rl is 
obtained by linear interpolation, as follows. 
9i = (92 - 91) /m * i + 9i 

As described above, according to the fourth 

10 embodiment, the linear interpolation is performed to 
the object observation directions of the panoramic 
images at the two points, whereby the annotation 
display position can be easily determined to the 
group of the panoramic images related to the route 

15 located between the two points. Moreover, the linear 
interpolation is used to obtain the annotation 
display position, whereby an amount of the 
calculation can be reduced. 
(Fifth Embodiment) 

2 0 In the above fourth embodiment, the annotation 

display position is obtained by performing the linear 
interpolation to the object observation directions of 
the panoramic images at the two points. On one hand, 
according to the fifth embodiment, the annotation 

25 display position is obtained more precisely by 
performing non-linear interpolation. 

Figs. 22A, 22B and 22C are diagrams for 
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explaining the relations of the object observation 
directions (angles) 91, 92 and 9i shown in Fig. 21. 
In each of Figs. 22A, 22B and 22C, the horizontal 
axis indicates frame numbers, and the vertical axis 
5 indicates object observation directions. Moreover, 
the range of the object observation direction 91 is 
limited to 0 ^ 91 < n, and the range of the object 
observation direction 92 is limited to 0 < 92 < n. 
Furthermore, it is assumed that the intervals of the 

10 frame taking positions on the route Rl are all equal. 
Here, Fig. 22A shows the object observation 
directions from the respective frames on the route Rl 
in case of n/2 < 91 < n and n/2 < 92 < n. Fig. 22B 
shows the object observation directions from the 

15 respective frames on the route Rl in case of 0 ^ 91 ^ 
n/2 and n/2 ^ 92 < n. Fig. 22C shows the object 
observation directions from the respective frames on 
the route Rl in case of 0 < 91 < n/2 and 0 < 92 < n/2. 
As shown in Figs. 22A to 22C, when it is assumed that 

20 the panoramic images are taken at the same intervals, 
the object observation directions do not change 
linearly but change non-linearly . Incidentally, each 
of the non-linear curves . shown in Figs. 22A to 22C 
corresponds to an arctangent function obtained by the 

25 object observation directions (angles) 91 and 92. 

When the annotation display position of the 
object is determined by the annotation display 
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position determination unit 50, the annotation 
display position (angle) 9i is determined by using, 
as an interpolation function, the arctangent function 
obtained from the object observation directions 
5 (angles) 91 and 92 from the two section points at 

both the ends of the route. Incidentally, to reduce 
an amount of calculation, a linearly approximated 
function of the arctangent function may be uses as 
the interpolation function. 

10 Moreover, to reduce an amount of calculation by 

the arctangent function, a table which indicates the 
relations between frame numbers and annotation 
display positions may be prepared beforehand. As 
shown in Figs. 23A, 23B, 23C, 23D, 23E, 23F and 23G, 

15 in case of -n < 91 < n and -n < 92 < n, the relations 
between the frame numbers and the annotation display 
positions are classified into six kinds (or patterns) 
of arctangent-function shapes in accordance with the 
object observation directions 91 and 92. The 

20 annotation display position determination unit 50 
holds beforehand the correspondence table which 
indicates six-pattern relations between the frame 
numbers and the annotation display positions based on 
representative values of the object observation 

25 directions 91 and 92, judges to which of the six 

patterns the target is closest on the basis of the 
object observation directions 91 and 92 from the 
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section points at both the ends of the current route, 
and refers to the value of the correspondence table 
on the basis of the corresponding frame number. It 
should be noted that it may be judged beforehand to 
5 which of the six patterns the target is closest. The 
correspondence table changes according to the number 
of panoramic images related to the route. Here, the 
correspondence table is formed beforehand with 
sufficiently fine resolutions, and the scale thereof 

10 is controlled according to the number of panoramic 

images on the corresponding route, whereby the number 
of correspondence tables is controlled. Incidentally, 
although the six correspondence tables are provided 
in the present embodiment, the number of 

15. correspondence tables can be increased according to 
the capacity of a RAM or the like. Moreover, in a 
case where the small number of correspondence tables 
are provided initially, when approximation cannot be 
achieved by the current correspondence tables in view 

20 of the actual annotation display positions , the 
interpolation functions determined by using the 
object observation directions from the section points 
at the both ends of the route to which the displaying 
is deviated or shifted are added as needed. By doing 

25 so, the accuracy can be increased. 

As described above, according to the fifth 
embodiment, the interpolation is performed by using 
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the arctangent functions obtained based on the object 
observation directions from the panoramic images at 
the two points, the annotation display positions can 
be determined more accurately to the group of the 
5 panoramic images related to the route located between 
the two points, 
(Other Embodiments ) 

Although the panoramic images are used in the 
above embodiments, images other than the panoramic 

10 image may be also used. 

Moreover, to provide program codes of software 
for achieving the functions of the above embodiments 
through a network is included in the concept of the 
present invention . 

15 In this case, the program codes themselves of 

software achieve the functions of the above 
embodiments, whereby the program codes themselves and 
a means for supplying the program codes to a computer 
constitute the present invention. 

20 Moreover, it is to be understood that the 

present invention includes not only the case where 
the functions of the above embodiments are achieved 
when the computer executes the supplied program codes 
but also a case where the functions of the above 

25 embodiments are achieved when the computer executes 
the supplied program codes in cooperation with an 
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operating system (OS) running on the computer, 
another application software or the like. 

As many apparently widely different embodiments 
of the present invention can be made without 
5 departing from the spirit and scope thereof, it is to 
be understood that the invention is not limited to 
the specific embodiments thereof except as defined in 
the appended claims . 



